Analytical Inductive Programming as a Cognitive Rule Acquisition Devise

One of the most admirable characteristic of the human cognitive system is its ability to extract generalized rules covering regularities from example experience presented by or experienced from the environment. Humans’ problem solving, reasoning and verbal behavior often shows a high degree of systematicity and productivity which can best be characterized by a competence level reflected by a set of recursive rules. While we assume that such rules are different for different domains, we believe that there exists a general mechanism to extract such rules from only positive examples from the environment. Our system Igor2 is an analytical approach to inductive programming which induces recursive rules by generalizing over regularities in a small set of positive input/output examples. We applied Igor2 to typical examples from cognitive domains and can show that the Igor2 mechanism is able to learn the rules which can best describe systematic and productive behavior in such domains. Introduction Research in inductive programming is concerned with the design of algorithms for synthesis of recursive programs from incomplete specifications such as input/output examples of the desired program behavior, possibly together with a set of constraints about size or time complexity (Biermann, Guiho, & Kodratoff 1984; Flener 1995). In general, there are two distinct approaches to inductive programming – search-based generate-and-test algorithms (Olsson 1995; Quinlan & Cameron-Jones 1995) and data-driven analytical algorithms (Summers 1977; Kitzelmann & Schmid 2006). In the first case, given some language restriction, hypothetical programs are generated, tested against the specification and modified until they meet some given criteria. In the second case, regularities in the input/output examples are identified and a generalized structure is built over the examples. While searchbased approaches – in principle – can generate each possible program and therefore might be able to find the ∗Research was supported by the German Research Community (DFG), grant SCHM 1239/6-1. Copyright c © 2008, The Second Conference on Artificial General Intelligence (AGI-09.org). All rights reserved. desired one given enough time, analytical approaches have a more restricted language bias. The advantage of analytical inductive programming is that programs are synthesized very fast, that the programs are guaranteed to be correct for all input/output examples and fulfill further characteristics such as guaranteed termination and being minimal generalizations over the examples. The main goal of inductive programming research is to provide assistance systems for programmers or to support end-user programming (Flener & Partridge 2001). From a broader perspective, analytical inductive programming provides algorithms for extracting generalized sets of recursive rules from small sets of positive examples of some behavior. Such algorithms can therefore be applied not only to input/output examples describing the behavior of some program but to arbitrary expressions. Taking this standpoint, analytical inductive programming provides a general device for the acquisition of generalized rules in all such domains where it is natural that people are typically exposed to only positive examples. This is, for example, the case in learning correct grammatical constructions where a child would never get explicitly exposed to scrambled sentences (such as house a is this). In the sixties, Chomsky proposed that the human mind possesses a language acquisition device (LAD) which allows us to extract grammar rules from the language experience we are exposed to (Chomsky 1959; 1965). Input to this device are the linguistic experiences of a child, output is a grammar reflecting the linguistic competence. The concept of an LAD can be seen as a special case of a general cognitive rule acquisition device. Unfortunately, this idea became quite unpopular (Levelt 1976): One reason is, that only performance and not competence is empirically testable and therefore the idea was only of limited interest to psycho-linguists. Second, Chomsky (1959) argued that there “is little point in speculating about the process of acquisition without much better understanding of what is acquired” and therefore linguistic research focussed on search for a universal grammar. Third, the LAD is concerned with learning and learning research was predominantly associated with Skinner’s reinforcement learning approach which clearly is unsuitable as a language acquisition device since it explains language acquisition as selective reinforcement of imitation. Since the time of the original proposal of the LAD there was considerable progress in the domain of machine learning (Mitchell 1997) and we propose that it might be worthwhile to give this plausible assumption of Chomsky a new chance. The conception of inductive biases (Mitchell 1997) introduced in machine learning, namely restriction (i.e. language) and preference (i.e. search) bias might be an alternative approach to the search of a universal grammar: Instead of providing a general grammatical framework from which each specific grammar – be it for a natural language or for some other problem domain – can be derived, it might be more fruitful to provide a set of constraints (biases) which characterize what kinds of rule systems are learnable by humans. Since we are interested in a mechanism to induce general, typically recursive, rules and not in classification learning, we propose to investigate the potential of analytical inductive programming as such a general rule acquisition device. Furthermore, we propose to take a broader view of Chomsky’s idea of an LAD and we claim that rule acquisition in that fashion is not only performed in language learning but in all domains where humans acquire systematic procedural knowledge such as problem solving and reasoning. In the following we give a short overview of our analytical inductive programming system Igor2 together with its biases. Then we illustrate Igor2’s ability as a cognitive rule acquisition device in the domains of problem solving, reasoning, and natural language processing. Recursive Structure Generalization Igor2 (Kitzelmann 2008) was developed as a successor to the classical Thesys system for learning Lisp programs from input/output examples (Summers 1977) and its generalization Igor1 (Kitzelmann & Schmid 2006). To our knowledge, Igor2 is currently the most powerful system for analytical inductive programming. Its scope of inducable programs and the time efficiency of the induction algorithm compares well with inductive logic programming and other approaches to inductive programming (Hofmann, Kitzelmann, & Schmid 2008). The system is realized in the constructor term rewriting system Maude Therefore, all constructors specified for the data types used in the given examples are available for program construction. Since Igor2 is designed as an assistant system for program induction, it relies on small sets of noise-free positive input/output examples and it cannot deal with uncertainty. Furthermore, the examples have to be the first inputs with respect to the complexity of the underlying data type. Given these restrictions, Igor2 can guarantee that the induced program covers all examples correctly and provides a minimal generalization over them. Classification learning The complete data sets and results can be found on www.cogsys.wiai.uni-bamberg.de/effalip/download.html. for noise-free examples such as PlayTennis (Mitchell 1997) can be performed as a special case (Kitzelmann 2008). Igor2 specifications consist of such a set of examples together with a specification of the input data type. Background knowledge for additional functions can (but needs not) be provided. Igor2 can induce several dependent target functions (i.e., mutual recursion) in one run. Auxiliary functions are invented if needed. In general, a set of rules is constructed by generalization of the input data by introducing patterns and predicates to partition the given examples and synthesis of expressions computing the specified outputs. Partitioning and search for expressions is done systematically and completely which is tractable even for relative complex examples because construction of hypotheses is data-driven. Igor2’s restriction bias is the set of all functional recursive programs where the outermost function must be either non-recursive or provided as background knowledge. Igor2’s built-in preference bias is to prefer fewer case distinctions, most specific patterns and fewer recursive calls. Thus, the initial hypothesis is a single rule per target function which is the least general generalization of the example equations. If a rule contains unbound variables on its right-hand side, successor hypotheses are computed using the following operations: (i) Partitioning of the inputs by replacing one pattern by a set of disjoint more specific patterns or by introducing a predicate to the right-hand side of the rule; (ii) replacing the right-hand side of a rule by a (recursive) call to a defined function where finding the argument of the function call is treated as a new induction problem, that is, an auxiliary function is invented; (iii) replacing sub-terms in the right-hand side of a rule which contain unbound variables by a call to new subprograms. Problem Solving Often, in cognitive psychology, speed-up effects in problem solving are modelled simply as composition of primitive rules as a result of their co-occurrence during problem solving, e.g., knowledge compilation in ACT (Anderson & Lebière 1998) or operator chunking in SOAR (Rosenbloom & Newell 1986). Similarly, in AI planning macro learning was modelled as composition of primitive operators to more complex ones (Minton 1985; Korf 1985). But, there is empirical evidence that humans are able to acquire general problem solving strategies from problem solving experiences, that is, that generalized strategies are learned from sample solutions. For example, after solving Tower of Hanoi problems, at least some people have acquired the recursive solution strategy (Anzai & Simon 1979). Typically, experts are found to have superior strategic knowledge in contrast to novices in a domain (Meyer 1992). There were some proposals to the learning of domain specific control knowledge in AI planning (Shell & Carbonell 1989; Shavlik 1990; Mart́ın & Geffner 2000). All these approaches proposed to learn cyclic/recursive